Projection-type video display device

ABSTRACT

In a projection-type video display device that displays an image by projecting the image onto a projection object starting from a desk surface, a display screen on which a position and a gesture operation of a user are reflected is displayed in a predetermined area. An illumination and a camera are disposed in the projection-type video display device installed on the desk surface, a gesture using user&#39;s fingers is detected, and, for example, a menu screen and an operation guide are displayed according to the gesture. The shadows of the fingers are photographed by the camera, and a gesture is detected. According to the gesture, the display screen is moved, rotated, divided, or enlarged/reduced, or a menu screen is displayed. The display direction is determined in consideration of the direction of sides of the desk.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/037,164, filed May 17, 2016, which is incorporated by reference as iffully set forth therein.

TECHNICAL FIELD

The present invention relates to a projection-type video display devicethat projects an image.

BACKGROUND ART

A technology for controlling of a display direction to be a direction inwhich a video can be easily seen by a user when a projection-typedisplay device projects the video onto a horizontal face such as a deskhas been devised.

In Patent Document 1, a technology has been disclosed in which aprojection-type display device disposed on the upper side of aprojection object includes an imaging unit, a user interface imagecapturing an operator's motion is acquired, and a display screen iscontrolled based on a detection result of the motion.

In addition, in Patent Document 2, a method has been disclosed in whicha display device disposed on the upper side of a projection objectrecognizes the position and the motion of a user and the shape of aprojection object and a display position, a display direction, and thelike are controlled in accordance therewith.

CITATION LIST Patent Document

Patent Document 1: JP 2009-64109 A

Patent Document 2: JP 2013-76924 A

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In the technologies disclosed in Patent Documents 1 and 2, it isintended to display a video on a projection object such as a desk thatis easily movable from a display device disposed on the upper side ofthe projection object, and there is a problem in usability such as thenecessity of movement of the display device according to the movement ofthe projection object.

In addition, in order to recognize the position, the operation, and thelike of a user, a point to be photographed for recognizing the operationis located far from an imaging unit of the display device disposed onthe upper side, and a point to be photographed is blocked by a part ofuser's body to make it difficult to photograph the point, whereby thereis a problem in the accuracy of the recognition.

Furthermore, in recognizing the position, the operation, and the like ofa user and improving the usability of the display device using a resultof the recognition, it is considered that there is room for furtherimprovement.

Solutions to Problems

In order to solve the problems described above, according to the presentinvention, there is provided a projection-type video display deviceincluding: a video projection unit that projects a video; at least onelighting unit that emits illumination light different from that of theprojected video onto a surface at least partly overlapping a videoprojection surface onto which the video projection unit projects thevideo; an imaging unit that performs imaging using reflected light ofthe illumination light; a detection unit that detects at least aposition at which an operator's finger is brought into contact with thevideo projection surface based on a captured image acquired by theimaging unit; and a display control unit that performs control such thatat least one partial video including the video projected by the videoprojection unit is changed based on a result of the detection acquiredby the detection unit.

In addition, the projection-type video display device may be installedon the video projection surface and project the video.

Effects of the Invention

According to the present invention, a projection-type video displaydevice having improved usability can be provided, and there is an effectof contributing to the wide use of the device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram that illustrates an example in which a video isprojected onto a desk using a projection-type display device disposed onthe desk.

FIG. 2 is a diagram that illustrates an example of the configuration ofa projection-type display device provided with an operation detectingfunction controlling a display screen using a gesture input.

FIG. 3 is a diagram that illustrates an example of a state in which auser performs an operation for a display screen of the projection-typedisplay device provided with the operation detecting function.

FIG. 4A is a diagram that illustrates the shapes of the shadows ofuser's fingers generated by two illuminations.

FIG. 4B is a diagram that illustrates the shapes of the shadow of user'sfingers generated by two illuminations.

FIG. 5 is a diagram that illustrates the shapes of shadows according touser's operation positions.

FIG. 6 is a diagram that illustrates a relation between a space betweena finger and an operation surface and the shape of a shadow.

FIG. 7 is a diagram that illustrates a determination of an approachingdegree using feature points.

FIG. 8 is a diagram that illustrates a determination of a contact pointusing feature points.

FIG. 9 is a diagram that illustrates a case where feature points are setto different positions.

FIG. 10 is a diagram that illustrates the shapes of shadows of a casewhere an operation is performed using a plurality of fingers.

FIG. 11A is a diagram that illustrates a determination of a pointingdirection using a contour line.

FIG. 11B is a diagram that illustrates a determination of a pointingdirection using a contour line.

FIG. 11C is a diagram that illustrates a determination of a pointingdirection using a contour line.

FIG. 12A is a diagram that illustrates a process flow of an operationdetecting method.

FIG. 12B is a diagram that illustrates a process flow of an operationdetecting method.

FIG. 13 is a diagram that illustrates an example of control according tothe approaching degree of a finger.

FIG. 14A is a diagram that illustrates an example of control accordingto a pointing direction.

FIG. 14B is a diagram that illustrates an example of control accordingto a pointing direction.

FIG. 14C is a diagram that illustrates an example of control accordingto a pointing direction.

FIG. 15 is a diagram that illustrates that a user is present near arectangular desk.

FIG. 16 is a diagram that illustrates a display direction determinedbased on a user's position and the shape of the edge of the rectangulardesk.

FIG. 17 is a diagram illustrating that a user is present near a circulardesk.

FIG. 18 is a diagram that illustrates a display direction determinedbased on user's position and the shape of the edge of the circular desk.

FIG. 19 is a diagram that illustrates an example in which a displayposition and a display direction are determined by detecting a pluralityof persons present near a rectangular desk.

FIG. 20 is a diagram that illustrates an example in which a displayposition and a display direction are determined by detecting a pluralityof persons present near a circular desk.

FIG. 21A is a diagram that illustrates an example of parallel movementusing a finger operation.

FIG. 21B is a diagram that illustrates an example of parallel movementusing a finger operation.

FIG. 22A is a diagram that illustrates an example of rotary movementusing a finger operation.

FIG. 22B is a diagram that illustrates an example of rotary movementusing a finger operation.

FIG. 23A is a diagram that illustrates an example of an operation ofincreasing the number of display screens using a finger operation.

FIG. 23B is a diagram that illustrates an example of an operation ofincreasing the number of display screens using a finger operation.

FIG. 24A is a diagram that illustrates an example of an operation ofenlarging a display screen using a finger operation.

FIG. 24B is a diagram that illustrates an example of an operation ofenlarging a display screen using a finger operation.

FIG. 25A is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 25B is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 26A is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 26B is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 27A is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 27B is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 28A is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 28B is a diagram that illustrates an example of an operation of aplurality of fingers.

FIG. 29 is a flowchart that illustrates an example of an operation whenvisible external light is abnormally blocked.

FIG. 30 is a diagram that illustrates an example of an operation guidedisplay.

FIG. 31A is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 31B is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 31C is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 31D is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 32A is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 32B is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 32C is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 32D is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 33A is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 33B is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 33C is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 33D is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 34A is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 34B is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 34C is a diagram that illustrates an example of an operation of amenu or a detailed information display.

FIG. 34D is a diagram that illustrates an example of an operation of amenu or a detailed information display.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

First Embodiment

FIG. 1 illustrates an example in which two videos 202 and 203 havingmutually-different rotation angles are projected onto a desk for aplurality of users facing mutually-different directions by using aprojection-type display device 201 disposed on the desk. In descriptionpresented above, while an example in which a video is projected onto a“desk” is described, the “desk” is an example of a video projectionsurface. Thus, this embodiment may have any other video projectionsurface (may be a surface or a floor of any other structure) as itstarget. By installing the projection-type display device 201 on thedesk, the adjustment of the position of the projection-type displaydevice 201 at the time of moving the desk is basically unnecessary.

In FIG. 1, a maximum range 210 in which the projection-type displaydevice 205 can optically project an image is illustrated using brokenlines. The maximum projection range 210 may be illustrated in drawingsdescribed below. For example, the projection-type display device 205projects two display screens 202 and 203 within the maximum projectionrange 210. The display screens 202 and 203 correspond to a screen of onscreen display (OSD) to be described later. In other words, imagesdisplayed on the display screens 202 and 203 are partial images withinthe maximum projection range 210.

For example, a way to use is considered in which, for a person presenton the periphery of the desk, a design diagram of a device is displayedin the whole maximum projection range 210, and an explanatory documentof the design diagram is displayed on the display screens 202 and 203.

FIG. 2 illustrates an example of a configuration diagram of aprojection-type display device 205 provided with an operation detectingfunction of which a display screen is controlled by a user through agesture input. The projection-type display device 205 provided with theoperation detecting function includes: a camera 100; two illuminations101 and 102; a shadow area extracting unit 104; a feature pointdetecting unit 105; an approaching degree detecting unit 106; a contactpoint detecting unit 107; a contour detecting unit 108; a directiondetecting unit 109; a control unit 110; a display control unit 111; adrive circuit unit 112; an input terminal 113; an input signalprocessing unit 114; and a projection unit 115. The control unit 110generates detection result data of operation states such as anapproaching degree of a finger with respect to an operation surface, thecoordinates of a contact point, a pointing direction, and the likedetected by the detection units.

In addition, the display control unit 111 generates display control datasuch as an operation mode, a pointer display position, a pointer displaydirection, and the like based on the detection results and performs aprocess based on the display control data for a video signal passingthrough the input terminal 113 and the input signal processing unit 114.A process that is necessary for the projection is performed by the drivecircuit unit 112 for a processed video signal, and a resultant videosignal is projected from the projection unit 115 to a projection object.In FIG. 2, while a buffer, a memory, and the like are not illustrated, abuffer, a memory, and the like that are necessary may be appropriatelymounted.

FIG. 3 is a diagram that illustrates an example of a state in which auser 3 performs an operation for a display screen 203 of aprojection-type display device 205 provided with the operation detectingfunction. The user 3 causes a finger 30, which is an operation unit, toapproach the display screen 203 of a projection object 204 and to be incontact with a certain position, thereby performing a desired operation.In other words, for example, the display screen 203 of the projectionobject 204 that is a desk surface is also an operation surface used forthe user 3 to perform an operation for the projection-type displaydevice 205. The operation performed by the user 3 will be describedlater in detail.

The projection-type display device 205 provided with the operationdetecting function includes two illuminations 101 and 102 and a camera100. The two illuminations 101 and 102 irradiate the finger 30 of theuser 3, and the camera 100 images the finger 30 and the proximitythereof. When the finger 30 approaches or is brought into contact withthe projection object 204, the shape of the shadow of the finger 30changes. Accordingly, the projection-type display device 205 analyses animage acquired by the camera 100, thereby detecting the approachingdegree, the contact point, and the pointing direction of the finger.

In addition, by installing the projection-type display device 205provided with the operation detecting function on the desk, a chance forblocking light when the illuminations 101 and 102 irradiate the finger30 can be reduced.

Next, an example of the operation of each unit of the projection-typedisplay device 205 provided with the operation detecting functionillustrated in FIG. 2 will be described. The camera 100 is configured byan image sensor, a lens, and the like and captures an image includingthe finger 30 that is the operation unit of the user 3. Each of the twoilluminations 101 and 102 is configured by a light emitting diode, acircuit board, a lens, and the like and projects the shadow of thefinger 30 within an image captured by the camera 100 by emittingillumination light to the projection object 204 and the finger 30 of theuser 3.

It may be configured such that each of the illuminations 101 and 102 isan infrared lamp, and the camera 100 is an infrared camera. In such acase, an infrared image captured by the camera 100 can be acquiredseparately form a visible-light video that is a video of a video signalprojected from the projection-type display device 205 provided with theoperation detecting function. In addition, illumination light andimaging light may be a non-visible ray that can be separated from aprojection video projected from the projection-type display device 205,and thus, ultraviolet light other than the infrared light may be used.

The shadow area extracting unit 104 generates a shadow image byextracting a shadow area from an image acquired from the camera 100. Forexample, it may be configured such that a differential image isgenerated by subtracting a background image of the projection object204, which has been captured in advance, from a captured image capturedat the time of detecting an operation, the luminance values of thedifferential image are binarized using a predetermined threshold Lth,and an area of which the luminance value is the threshold or less is setas a shadow area. In addition, a so-called labeling process in whichshadow areas not connected to each other among extracted shadows areidentified as different shadows is performed. According to the labelingprocess, fingers to which a plurality of the extracted shadowscorrespond can be determined, in other words, two shadows forming a paircorresponding to one finger can be identified.

The feature point detecting unit 105 detects a specific position(hereinafter referred to as a feature point) within a shadow imageextracted by the shadow area extracting unit 104. For example, as thefeature point, a tip end position (corresponding to a fingertipposition) within a shadow image is detected. In order to detect afeature point, while various techniques are used, in case of a tip endposition, the tip end position can be detected from coordinate data ofpixels configuring the shadow image, or a portion matching a uniqueshape of the feature point can be detected through image recognition orthe like. Since one feature point is detected from one shadow, twofeature points are detected for one finger (two shadows).

The approaching degree detecting unit 106 measures a distance d betweentwo feature points detected by the feature point detecting unit 105 anddetects a space s (approaching degree A) between the finger and theoperation surface based on the distance d. Accordingly, it is determinedwhether or not the finger is brought into contact with the operationsurface.

In a case where the finger is determined to be brought into contact withthe operation surface by the approaching degree detecting unit 106, thecontact point detecting unit 107 detects a contact point of the fingerfor the operation surface based on the position of the feature point andcalculates the coordinates thereof.

The contour detecting unit 108 extracts the contour of a shadow areafrom the shadow image extracted by the shadow area extracting unit 104.For example, a start pixel of contour tracing is determined by scanningthe inside of the shadow image in a constant direction, and neighboringpixels of the start pixel are traced in a counterclockwise direction,whereby the contour is acquired.

The direction detecting unit 109 extracts segments each having an almostlinear shape from the contour line detected by the contour detectingunit 108. Then, based on the direction of the extracted contour line,the pointing direction of the finger on the operation surface isdetected.

The process of each detection unit described above is not limited to thetechnique described above, but an algorithm of another image processingmay be used. The detection unit described above may be configured notonly by hardware using a circuit board but also by software.

The control unit 110 generates detection result data such as theapproaching degree, the coordinates of a contact point, the pointingdirection, and the like of the finger for the operation surface, whichare detected by the detection units, by controlling the overalloperation of the device.

The display control unit 111 generates display control data such as anoperation mode, a pointer position, a pointer direction, and the likebased on the detection result data such as the approaching degree, thecoordinates of the contact point, the pointing direction, and the likeof the finger generated by the control unit 110 and performs a processbased on the display control data for a video signal passing through theinput terminal 113 and the input signal processing unit 114.

The drive circuit unit 112 performs a process used for projecting theprocessed video signal as a display video. The display image isprojected onto the projection object from the projection unit 115.

While the units described up to here have been described to be equippedin one projection-type display device 205 provided with the operationdetecting function as an example, some thereof may be configured asseparate units and be connected using a transmission line.

FIGS. 4A and 4B are diagrams that illustrate the shapes of shadows of auser's finger generated by two illuminations. FIG. 4A illustrates astate in which the finger 30 and the projection object 204 are notbrought into contact with each other, and FIG. 4B illustrates a state inwhich the finger and the projection object are brought into contact witheach other.

As illustrated in FIG. 4A, in the state (space s) in which the finger 30is not brought into contact with the projection object 204, light fromtwo illuminations 101 and 102 is blocked by the finger 30, and shadows401 and 402 (denoted by diagonal lines) are formed. In a camera image,the two shadows 401 and 402 are present to be separate to both sides ofthe finger 30.

On the other hand, as illustrated in FIG. 4B, in the state (space s=0)in which the fingertip of the finger 30 is brought into contact with theprojection object 204, two shadows 401 and 402 are present to approacheach other at the position of the fingertip of the finger 30. Whilepartial areas of the shadows 401 and 402 are hidden in the shade of thefinger 30, the hidden portions are not included in the shadow areas. Inthis embodiment, a contact between the finger 30 and the projectionobject 204 is determined by using a property of a space (particularly, adistance between feature points) between the shadow 401 and the shadow402 approaching each other when the finger 30 approaches the projectionobject 204.

FIG. 5 is a diagram that illustrates the influence of user's operationon shapes of shadows. Here, a camera image of a case where the user'soperation position deviates to the left side from the center of theprojection object 204 (a user position 3) and a camera image of a casewhere the user's operation position deviates to the right side (userposition 3′) are compared with each other. At this time, while theuser's operation position viewed from the camera 100 changes, in suchcamera images, the positional relations of shadows 401 (401′) and 402(402′) with respect to the finger 30 (30′) are not changed. In otherwords, regardless of the user's operation position, the shadows 401(401′) and 402 (402)′ are constantly present on both sides of the finger30 (30′). The reason for this is that the positional relations areuniquely determined based on the positional relations between the camera100 and the illuminations 101 and 102. Accordingly, even in a case wherethe user performs an operation at any position with respect to theprojection object 204, two shadows 401 and 402 can be detected, and theoperation detecting method according to this embodiment can beeffectively applied.

FIG. 6 is a diagram that illustrates a relation between a space betweena finger and an operation surface and the shape of a shadow. A gapbetween two shadows 401 and 402 formed on both sides of the finger 30changes according to a space s between the finger 30 and the projectionobject 204. In order to define the gap between the two shadows 401 and402, feature points 601 and 602 (denoted by x marks) are respectivelyset on the inside of the shadows 401 and 402, and a distance d betweenthe feature points is measured. Here, the feature points are set at thetip end positions (fingertip positions) of the shadows. In a case wherethe space s between the finger 30 and the projection object 204 islarge, the gap between the two shadows 401 and 402 is large, and thedistance d between the two feature points 601 and 602 is large as well.As the finger 30 approaches the projection object 204, the distance dbetween the feature points 601 and 602 is decreased, and, when thefinger 30 is brought into contact with the projection object 204 (spaces=0), the distance d between the feature points 601 and 602 is a minimalvalue.

FIG. 7 is a diagram that illustrates a determination of an approachingdegree performed by the approaching degree detecting unit 106. Here, adetermination of the approaching degree A of the finger is performedbased on a distance d between feature points. In order to determine theapproaching degree A of the finger, four thresholds d1, d2, d3, and d4(here, d1<d2<d3<d4) are set for the distance d between feature points.In this way, the approaching degree A is classified into five levels(levels 1 to 5), and, the larger the level value is, the smaller thespace s between the finger 30 and the projection object 204 is. First,the threshold d1 used for identifying a state (space s=0) in which thefinger 30 is brought into contact with the projection object 204 is set,and, in a case where the distance d<d1, the approaching degree A isdetermined to be a maximum level 5 (contact state). Then, for the othernon-contact states, the approaching degree A is classified into fourlevels (levels 4 to 1) by using the thresholds d2 to d4. Among these, ina case where d>d4, the approaching degree A is determined to be aminimum level 1. In this example, while the approaching degree isclassified into the five levels by using the four thresholds, theclassified number of the approaching degrees is not limited thereto butmay be appropriately set according to a control content.

FIG. 8 is a diagram that illustrates a determination of a contact pointperformed by the contact point detecting unit 107. The shapes of theshadows 401 and 402 in the state in which the finger 30 is brought intocontact with the projection object 204 are illustrated, and, here,feature points 601 and 602 are respectively set as tip end positions ofshadows 401 and 402. In this case, since the two feature points 601 and602 are close to the position of a fingertip that is a contact point, amiddle point P between the two feature points 601 and 602 is regarded asa contact point between the finger 30 and the projection object 204, andthe coordinates thereof can be calculated.

In the example described above, while the feature points 601 and 602 arerespectively set to the tip end positions of the shadows 401 and 402,according to this method, feature points can be easily set, and theposition of the contact point P is present neighboring thereto and thus,can be easily determined.

FIG. 9 is a diagram that illustrates a case where feature points are setto different positions. While the feature points 601 and 602 are set tothe tip end positions of the shadows 401 and 402 in the case illustratedin FIG. 8, feature points 601′ and 602′ are set to middle positions ofshadows in the longitudinal directions in the case illustrated in FIG.9. Also in this case, a distance d′ between the feature points 601′ and602′ changes in accompaniment with a change in the gap between theshadows 401 and 402, and accordingly, a determination of the approachingdegree A between the finger 30 and the projection object 204 can beperformed. Since the contact point P′ of this case deviates from thepositions of the feature points 601′ and 602′ in the longitudinaldirection, a distance (correction amount) to the contact point P′ thatis predicted based on the feature points 601′ and 602′ is acquired inadvance, and a correction is performed using the distance, whereby thecontact point P′ can be acquired. Similarly, feature points may be setto positions other than positions disposed inside the shadows 401 and402.

FIG. 10 is a diagram that illustrates the shapes of shadows of a casewhere an operation is performed using a plurality of fingers. When aplurality of fingers 31, 32, . . . are brought into contact with theoperation surface in a hand-open state, left shadows 411, 421, . . . andright shadows 412, 422, . . . are formed for the fingers. Then, afeature point is set for each of the shadows. Here, feature points 611and 612 for the shadows 411 and 412 and feature points 621 and 622 forthe shadows 421 and 422 are illustrated. By measuring a distance dbetween the corresponding feature points 611 and 612 or thecorresponding feature points 621 and 622, an approaching degree or acontact point of each of the fingers 31 and 32 can be acquired. In thisway, according to this embodiment, contacts for the plurality of fingerscan be independently detected also in the hand-open state, andaccordingly, the embodiment can be applied to a multi-touch operation.

FIGS. 11A to 11C are diagrams that illustrate determinations of pointingdirections that are performed by the direction detecting unit 109. Inthe drawings, the shapes of the shadows 401 and 402 when the direction(pointing direction) of the finger 30 is inclined are illustrated, andthe directions of the shadows 401 and 402 are changed in accompanimentwith a change in the pointing direction. In order to detect the pointingdirection, first, the contour detecting unit 108 detects contour lines501 and 502 of the shadows 401 and 402. In the detection of each contourline, a curved portion such as a fingertip is eliminated, and a contourline formed by segments each having an approximately linear shape isdetected. Thereafter, the direction detecting unit 109 determines apointing direction using the following method.

In the case illustrated in FIG. 11A, inner contour lines 501 and 502 ofthe shadows 401 and 402 are used. Then, one of the inclining directions701 and 702 of the inner contour lines 501 and 502 is determined as thepointing direction.

In the case illustrated in FIG. 11B, outer contour lines 501′ and 502′of the shadows 401 and 402 are used. Then, one of the incliningdirections 701′ and 702′ of the outer contour lines 501′ and 502′ isdetermined as the pointing direction.

In the case illustrated in FIG. 11C, the inner contour lines 501 and 502of the shadows 401 and 402 are used. Then, the inclining direction 703of a median line of the inner contour lines 501 and 502 is determined asa pointing direction. In this case, since the pointing direction isacquired based on an average of the directions of the two contour lines501 and 502, the accuracy is improved. Here, the direction of a medianline of the outer contour lines 501′ and 502′ may be determined as thepointing direction.

FIGS. 12A and 12B are diagrams that illustrate process flows of anoperation detecting method according to Embodiment 1. FIG. 12A is aflowchart that illustrates the detection of an approaching degree and acontact point, and FIG. 12B is a flowchart that illustrates thedetection of a pointing direction.

First, the method of detecting an approaching degree and a contact pointillustrated in FIG. 11A will be described.

In S1001, the shadow area extracting unit 104 acquires a differentialimage by subtracting a background from an image captured by the camera100 and extracts a portion of which the luminance is the threshold Lthor less as a shadow area. At that time, a so-called labeling process inwhich shadow areas not connected to each other among extracted shadowsare identified as different shadows is performed.

In S1002, the feature point detecting unit 105 detects a feature pointfor each labeling-processed shadow. For example, as illustrated in FIG.6, the tip end positions of the shadows 401 and 402 are detected asfeature points 601 and 602.

In S1003, a distance d between the two feature points 601 and 602 thathave been detected is measured.

In S1004, the approaching degree detecting unit 106 determines anapproaching degree A between the finger 30 and the projection object 204based on the distance d. In the determination, for example, the distanced is compared with the thresholds d1 to d4 with reference to FIG. 7 andclassifies the approaching degree A into the levels 1 to 5. Then, in acase where d<d1, it is determined that the approaching degree A=5(contact state).

In S1005, it is determined whether or not the determined approachingdegree A is the level (=5) at the time of a contact. In a case where aresult of the determination is the approaching degree A=5, the processproceeds to S1006. On the other hand, in case of another determinationresult (non-contact state), the process ends.

In S1006, the contact point detecting unit 107 detects a contact pointbetween the finger 30 and the projection object 204. For example, asillustrated in FIG. 8, a middle point P of two feature points 601 and602 is determined as a contact point, and the coordinates thereof arecalculated. In a case where the method of setting feature points isdifferent from that described above (tip end position), the position ofthe contact point may be corrected according to the setting method.

In an operated state, the process flow described above is repeatedlyexecuted, and operation detection following a change in the operationstate is performed.

Next, a method of detecting the pointing direction illustrated in FIG.12B will be described.

In S1011, the shadow area extracting unit 104 acquires a differentialimage by subtracting a background from an image captured by the camera100 and extracts a portion of which the luminance is the threshold Lthor less as a shadow area. This is similar to S1001 described above.

In S1012, the contour detecting unit 108 detects a contour line (anapproximately linear portion) for each labeling-processed shadow. Forexample, as illustrated in FIG. 11C, the inner contour lines 501 and 502of the shadows 401 and 402 are detected. At that time, a curved portionsuch as a fingertip is eliminated from such a contour line, wherebysegments each having an approximately linear shape are detected.

In S1013, the direction detecting unit 109 determines the incliningdirection 703 of a median line of the contour lines 501 and 502 as thepointing direction. For the determination of the pointing direction, amethod illustrated in FIG. 11A or FIG. 11B may be used.

In an operated state, the process flow described above is repeatedlyexecuted, and operation detection following a change in the operationstate is performed.

In addition, the process of detecting an approaching degree and acontact point illustrated in FIG. 12A and the pointing directiondetecting process illustrated in FIG. 12B may be performed in parallel.

FIG. 13 is a diagram that illustrates an example of control according tothe approaching degree of a finger performed by the display control unit121.

In the diagram, switching among operation modes and pointer displayswitching according to an approaching degree A between the finger 30 andthe projection object 204 are illustrated.

At the level 5 (contact state), which is the highest level, of theapproaching degree A, the operation mode is set to a contact operationmode. At the levels 4 and 3, which are relatively high, of theapproaching degree A in the other non-contact states, the operation modeis switched to an air operation mode. In addition, at the levels 2 and1, which are relatively low, of the approaching degree A, the operationmode is switched to an operation-off mode. According to such control,also in a state in which the finger is floating from the projectionobject 204 in addition to the state in which the finger 30 is broughtinto contact with the projection object 204, the user 3 can operate anoperation target device. In addition, in a case where the finger 30 isseparate from the projection object 204 by a predetermined distance ormore, the operation mode is switched to the operation-off mode, and anunintended user's operation can be prevented.

In addition, regarding the pointer display, at the levels 5 and 4, whichare relatively high, of the approaching degree A, the pointer isdisplayed, and, at the levels 3, 2, and 1, which are relatively low, ofthe approaching degree A, the pointer is switched not to be displayed.According to such control, the user 3 can check the pointer in a stagebefore the finger 30 being brought into contact with the projectionobject 204, and the position of the pointer at the time of a contact canbe easily matched. According to the control described above, theoperability of the operation target device is improved.

FIGS. 14A to 14C are diagrams that illustrate an example of controlaccording to a pointing direction performed by the display control unit121.

FIG. 14A illustrates a case where the display position of the pointer800 is corrected according to the pointing direction 700. When a pointer800 is displayed, in a case where the pointer is displayed at a positionthat is exactly the same as that of a contact point P detected by thecontact point detecting unit 107, the pointer 800 is hidden by thefinger 30 and, it is difficult for the pointer to be seen from the user3. Thus, along the pointing direction 700 detected by the directiondetecting unit 109, the pointer 800 is displayed to be shifted to thefront side of the fingertip by a predetermined amount. Accordingly, thepointer 800 can be easily seen from the user 3.

In addition to the description presented above, the shift amount(correction amount) of the display position of the pointer 800 may bechanged according to the approaching degree A of the finger 30. Forexample, when the approaching degree A is low, the correction amount isincreased, and, when the approaching degree A is high, the correctionamount is decreased. Accordingly, as the finger 30 of the user 3approaches the projection object 204, the position of the pointer 800becomes closer to the fingertip, and the user 3 can perform an operationhaving high accuracy by using the display position of the pointer 800 asthe reference.

FIG. 14B illustrates a case where the display direction of the pointer800 is corrected according to the pointing direction 700. When a pointer800 having an arrow shape is displayed, for example, as illustrated inFIG. 14A, in a case where the pointer is constantly displayed in aconstant direction regardless of the pointing direction 700, there arecases where the display direction of the pointer 800 does not match thepointing direction 700, and the user 3 is caused to feel discomfort.Thus, the direction detecting unit 109 performs display such that thedisplay direction of the pointer 800 matches the pointing direction 700detected by the direction detecting unit 109. In this way, there is nomismatching with the pointing direction 700, and the discomfort of theuser 3 can be resolved.

FIG. 14C illustrates a case where the position of the contact point iscorrected according to the pointing direction 700. The contact point Pdetected by the contact point detecting unit 107 is determined based onthe position of a feature point, and accordingly, there are cases wherethe contact point deviates from a position that is actually in contactwith the projection object 204. For example, in a case where the featurepoint is set to the tip end position of the shadow, the contact pointdeviates from an actual contact position (in many cases, a fingercushion portion) to a position of the tip end side (nail tip) of thefinger. Thus, along the pointing direction 700 detected by the directiondetecting unit 109, the position of the contact point is corrected tothe root side of the finger by a predetermined amount (P→P″).Accordingly, the contact point between the finger 30 and the projectionobject 204 can be more accurately acquired.

The method of detecting the operation content of the user 3 throughpointing detection or the like in the projection-type display device 205provided with the operation detecting function has been described asabove. In the detection system for a contact point and a pointingdirection through a pointing gesture according to the system describedabove, an operation can be performed in a case where a thin object suchas a finger is present. Compared to a light emitting pen type in which arecognition process is performed by emitting predetermined light from apen tip, in this system, a dedicated light emitting pen, or the likedoes not need to be secured, and the usability thereof is greatly high.

Subsequently, a screen operation realized by the operation of the user 3such as the pointing operation described until now will be described.

The number of display screens before the operation, the orientation ofthe display, the position of the display, the size of the display, andthe like are determined using a method presented in the followingexamples and the like.

For example, a setting of the display screen that is set as default inthe projection-type display device 205 provided with the operationdetecting function may be used.

In addition, the user may manually determine the number of displayscreens, the orientation of the display, the size of the display, andthe like by setting conditions.

Furthermore, through the pointing detection or the like described above,for example, by specifying the position at which a person is present byestimating the stretching direction of a finger or an arm or specifyingthe number of users based on the number of fingers or hands that arestretched forward, the display direction that can be easily recognizedor the number of display screens may be set.

In addition, the number and the positions of users, the shape of aprojection object, and the like are recognized, and the number ofdisplay screens, the display position, the orientation of the display,the size of the display, and the like may be determined in accordancetherewith. In the recognition of the number and the positions of users,the shape of the projection object, and the like, the projection-typedisplay device installed on a desk has advantages in that a distance toa recognition object is short, and the frequency at which therecognition object is blocked by an obstacle is low.

FIGS. 15 to 18 illustrate examples of methods for determining thedisplay direction of a display screen by recognizing the shape of aprojection object and the like. FIGS. 15 and 16 are examples of a casewhere projection is performed for a rectangular desk. As illustrated inFIG. 15, it is photographed and recognized by the camera 100 of theprojection-type display device 205 provided with the operation detectingfunction that a user 3 is present near the projection object 204 that isthe rectangular desk. In addition, the position of the edge of the deskof a most approaching portion 302 between the user 3 and the edge of thedesk is recognized. The display direction is determined to be adirection in which the direction of the edge of the position of the mostapproaching portion 302 and the bottom side of a display video areparallel to each other, and the position of the most approaching portion302 is on the lower side, in other words, a direction in which a displayvideo 202 illustrated in FIG. 16 is displayed.

FIGS. 17 and 18 are examples of a case where projection is performed fora circular desk. As illustrated in FIG. 17, it is photographed andrecognized by the camera 100 of the projection-type display device 205provided with the operation detecting function that a user 3 is presentnear the projection object 204 that is the rectangular desk. Inaddition, the position of the edge of the desk of a most approachingportion 303 between the user 3 and the edge of the desk is recognized.The display direction is determined to be a direction in which thedirection of the edge of the position of the most approaching portion303 and the bottom side of a display video are parallel to each other,and the position of the most approaching portion 303 is on the lowerside, in other words, a direction in which a display video 202illustrated in FIG. 18 is displayed.

In the description presented with reference to FIGS. 15 to 18, while theposition of the user 3 and the shape of a projection object aredescribed to be photographed using the camera 100 as an example, thecamera 100 is basically used for photographing an operation of the user3 through the pointing detection or the like, and accordingly, anadditional camera used for photographing the position of the user 3 andthe shape of a projection object may be provided.

In a case where a plurality of videos are displayed as illustrated inFIG. 3, and each display direction may be determined by performing asimilar operation for the positions of a plurality of persons. FIGS. 19and 20 illustrate examples of a case where a plurality of display videosare displayed. In any one of the examples, a plurality of persons andthe positions thereof are detected, and the display position and thedisplay direction are determined based on the position and the shape ofan edge of the desk that is closest from the position.

In this way, a display direction can be automatically determined basedon the shape of the edge of the desk that is closest to the user 3.

Examples of a case where a display screen determined using such a methodis operated according to a screen operation performed through a gestureof the user 3 are illustrated in FIGS. 21A and 24B. It is assumed that afingertip illustrated in each diagram is brought into contact with theprojection object 204. For example, FIG. 21A illustrate a state beforethe operation, and 21B illustrates a state after the operation. FIGS.22A to 24B similarly illustrate such states.

In many of the drawing of FIG. 21A and subsequent drawings, similar toFIG. 1, a maximum range 210 in which the projection-type display device205 can optically project an image is illustrated using broken lines. Asdescribed above, display screens 202 and 203 are displayed within thedisplay range of the maximum projection range 210 like an OSD. In otherwords, the display screens 202 and 203 are partial images in the maximumprojection range 210.

FIGS. 21A and 21B illustrate an example of a parallel movement. A fingerthat is brought into contact with the display screen 203 illustrated inFIG. 21A is moved in one of a vertical direction, a horizontaldirection, and an inkling direction without changing the direction.Then, as illustrated in FIG. 21B, only a display screen, with which thefinger is brought into contact, of the display screens is moved in asame manner as the movement of the finger. In this way, one of thedisplay screens can be moved to a position desired by the user.

FIGS. 22A and 22B illustrate an example of a rotary movement. A fingerthat is brought into contact with the display screen 203 illustrated inFIG. 22A is rotated. Then, as illustrated in FIG. 22B, the displaydirection of only a display screen, with which the finger is broughtinto contact, of display screens is rotated according to the motion ofthe finger. In this way, one of the display screens can be rotated in adirection desired by the user.

In the configuration of this embodiment, the pointing direction can bedetected. Accordingly, as illustrated in FIG. 22B, also in a case wherethe direction of a finger is rotated without changing the position of acontact point, a rotation operation of the display screen can berealized. This is the rotation operation that is difficult to berealized in a touch sensor of a tablet terminal or the like and can berealized first by employing the configuration of this embodiment.

When the rotation operation of a display screen is performed, there arecases where the display screen cannot be rotated with the size of thedisplay screen before the rotation maintained according to a relationbetween a shape within an optical projectable range of theprojection-type display device 205 and the position and the shape of thedisplay screen to be rotated. In such cases, a display screen reductionprocess may be performed in parallel in the middle of the rotation. In acase where the display screen reduction process is necessary when thedisplay screen is rotated in one direction, after the the display screenis rotated halfway and is reduced, when the display screen is rotated inthe opposite direction and is returned to the original angle, thedisplay screen may be returned to the original size by performing anenlargement process.

FIGS. 23A and 23B illustrate an example of an operation for increasingthe number of display screens. In the state illustrated in FIG. 23A, bymoving a finger brought into contact with the outside of the displayscreen 200 to the outside of the opposite side, a motion like a motionof cutting the display screen into halves is performed. Then, asillustrated in FIG. 23B, since the display screen is cut into halves,like cell division, two screens displaying the same content as thatdisplayed on the display screen 200 are generated (display screens 202and 203). In this way, the number of display screens as desired by theuser can be increased.

FIGS. 24A and 24B illustrate an example of an operation of enlarging adisplay screen. Two fingers brought into contact with the display screen202 illustrated in FIG. 24A are positioned at vertexes of a rectanglefacing each other, and a distance between the two fingers is increasedsuch that a diagonal line joining the vertexes facing each other isenlarged as illustrated in FIG. 24B. Then, the screen is enlargedcorresponding to the expansion of only the operated display screen 202.By operating in the opposite direction, an operation of displaying anarea that has run off to the outside of the display screen 202 until nowby reducing the screen can be performed. In this way, the display screencan be enlarged or reduced as is desired by the user.

Next, an example of a display screen operation using a plurality offingers will be described with reference to FIGS. 25A to 28B.

FIGS. 25A and 25B illustrate an example of rotating a display screen 203by operating two fingers. As illustrated in FIG. 25A, in a case whereboth two fingers are brought into contact with a desk within the displayscreen 203, when it is detected that the two fingers move such that aslope of a straight line joining contact points of the two fingerschanges as illustrated in FIG. 25B, the projection-type display device205 provided with the operation detecting function changes the displayangle of the display screen 203 so as to respond to the change in theslope. In this way, the rotation process of the display screen can beperformed.

In contrast to this, as illustrated in FIG. 26A, even in a case whereboth two fingers are brought into contact with the desk, in a case whereboth two contact points of the two fingers are not present inside onedisplay screen (for example, a case where only the contact point of onefinger is present within the display screen 203, and the contact pointof the other finger is present outside the display screen 203 or thelike), as illustrated in FIG. 26B, even when the two fingers move suchthat a slope of a straight line joining the contact points of the twofingers is changed, the display angle of the display screen 203 may notbe changed.

Among a plurality of fingers detected by the camera 100, two fingers ofwhich desk contact times have a difference less than a predeterminedtime may be determined by the control unit 110 as a combination offingers to be targets for the process described with reference to FIGS.25A to 26B. For example, among a plurality of fingers detected by thecamera 100, two fingers of which desk contact times have a differenceless than one second are detected by the control unit 110, and thecombination of the two fingers may be determined as a combination offingers that are targets for the process described with reference toFIGS. 25A to 26B.

As the contact positions of the two fingers described above and thecontact positions of the two fingers used in the process of determiningthe range of the display screen 203, first positions at which the twofingers are brought into contact with the desk from the air may be used.In a case where such fingers move from the outside of the display screen203 to the inside thereof while being brought into contact with the deskor the like, the rotation process of the display screen may not bestarted. In this way, by determining whether or not the rotation processis started based only on first positions at which the two fingers arebrought into contact with the desk from the air, the process can besimplified, and there is an effect of improving the process efficiencyof the control unit 110. In addition, according to the determinationprocess described above, among a plurality of display screens, a displayscreen that is a target for the rotation process operation can beclearly specified.

Next, FIGS. 27A and 27B illustrate an example in which a display screen202 is enlarged by operating two fingers. As illustrated in FIG. 27A, ina case where both two fingers are brought into contact with a deskinside the display screen 202, when it is detected that the two fingersmove such that a distance of a straight line joining contact points ofthe two fingers is increased as illustrated in FIG. 27B, theprojection-type display device 205 provided with the operation detectingfunction changes the display size of the display screen 202 so as torespond to the change in the length. In this way, the enlargementprocess of the display screen can be performed. On the other hand, in acase where the distance of the straight line joining the contact pointsof two fingers is decreased, a reduction process of the display screenis performed.

In contrast to this, as illustrated in FIG. 28A, even in a case whereboth two fingers are brought into contact with the desk, in a case whereone of the contact points of the two fingers is not present within onedisplay screen (for example, in a case where only the contact point ofone finger is present within the display screen 202, and the contactpoint of the other finger is present outside the display screen 202 orthe like), as illustrated in FIG. 28B, even when the two fingers movesuch that the length of the straight line joining the contact points ofthe two fingers is changed, the size of the display screen 202 may notbe changed.

Similar to the examples illustrated in FIGS. 25A to 26B, among aplurality of fingers detected by the camera 100, two fingers of whichdesk contact times have a difference less than a predetermined time maybe determined by the control unit 110 as a combination of fingers to betargets for the process described with reference to FIGS. 27A to 28B.For example, among a plurality of fingers detected by the camera 100,two fingers of which desk contact times have a difference less than onesecond are detected by the control unit 110, and the combination of thetwo fingers may be determined as a combination of two fingers that aretargets for the process described with reference to FIGS. 27A to 28B.

As the contact positions of the two fingers described above and thecontact positions of the two fingers used in the process of determiningthe range of the display screen 202, first positions at which the twofingers are brought into contact with the desk from the air may be used.In a case where such fingers move from the outside of the display screen202 to the inside thereof while being brought into contact with the deskor the like, the size changing process of the display screen may not bestarted. In this way, by determining whether or not the size changingprocess is started based only on first positions at which the twofingers are brought into contact with the desk from the air, the processcan be simplified, and there is an effect of improving the processefficiency of the control unit 110. In addition, according to thedetermination process described above, among a plurality of displayscreens, a display screen that is a target for the size changing processoperation can be clearly specified.

According to the display screen operation using a plurality of fingersdescribed with reference to FIGS. 25A to 28B as above, a display screenthat is a target for the rotation process or the size changing processcan be clearly specified. In addition, the process of the control unitcan be efficiently performed.

As described above, according to the projection-type display deviceprovided with the operation detecting function that can be installed ona desk, an operation according to pointing detection and the like can beperformed with high accuracy.

Second Embodiment

Next, Embodiment 2 of the present invention will be described.

The projection-type display device provided with the operation detectingfunction illustrated in FIG. 2 includes the illuminations 101 and 102,and the method for detecting shadows of illumination light emitted fromtwo different directions has been described in Embodiment 1. In a casewhere visible light is used as the illumination light, shadows generatedby a pointing operation are reflected on the display screen, and thereis a possibility that the visibility of the display screen is degraded.Thus, it is preferable that illuminations of non-visible light such asinfrared light as the illumination light are used, and a correspondingcamera such as an infrared camera is used as the camera 100.

As points to be noted in a case where non-visible light such as infraredlight is used as above, the non-visible light is not visible to theeyes, and thus, in a case where the illumination light is blocked byplacing an obstacle or the like before the illuminations 101 and 102,there are problems in that pointing detection or the like is notperformed, and the user does not notice the blockage.

In order to avoid such problems, by including a function for warning theuser in a case where a situation such as a blockage of the pointingdetection is detected by the camera 100, such problems can be solved.

FIG. 29 illustrates a flowchart of a case where such a function isoperated.

In S2001, it is assumed that the situation described above occurs. InS2002, the situation is photographed using the camera 100. In S2003, thecontrol unit 110 determines that pointing detection cannot be performedbased on an image captured by the camera and notifies the displaycontrol unit 111 of the determination.

As criteria for the detection, for example, there are long-timereflection of an object other than a detection object such as a fingerset in advance, the blockage of non-visible light over a wide range,reflection of an immovable object for a long time, and the like.

In S2004, the display control unit 111 generates a video acquiring byinserting a warning subtitle such as “Pointing detection is abnormal.Please check that there is no foreign material between a camera and alighting unit and a display screen.” into a display screen and displaysthe generated video in S2005. Until the camera 100 photographs asituation in which an obstacle of the non-visible light is removed, thisdisplay is continued.

In S2006, the control unit 110 determines whether or not the obstacle ofthe non-visible light is removed based on an image captured by thecamera 100. When the obstacle of the light is removed, in S2007, thecontrol unit 110 notifies the display control unit 111 that the pointingdetection function has been restored.

In S2008, the display control unit 111 generates a normal display screenin which the warning subtitle is eliminated and returns the displayscreen to the normal display screen in S2009. As above, an example of aseries of operations has been described.

In the description presented until now, while the method of notifyingthe user of an abnormality such as blockage of non-visible lightdisturbing the pointing detection or the like by inserting a warningsubtitle into the display screen has been described, a warning usinganother means such as a warning using a sound, lighting of a warninglamp arranged in advance, or the like may be used.

As described above, by including the function for detecting thesituation in which the pointing detection or the like is disturbed andthe function for warning the user, it can be prevented that functions ofthe pointing detection and the like are interrupted while the user doesnot notice the interrupt.

Third Embodiment

Next, Embodiment 3 of the present invention will be described.

In the projection-type display device described in Embodiment 1 of thepresent invention, as described above, various screen operations can beperformed through viewer's gesture operations (pointing operations).However, some of operations that can be input to the projection-typedisplay device according to this embodiment are operations that cannotbe performed in a finger touch detecting-type tablet and the like, andthere are also cases where a user is inexperienced. Thus, in aprojection-type display device according to Embodiment 3 of the presentinvention, in addition to the configuration and the functions of theprojection-type display device according to Embodiment 1 or Embodiment2, an operation guide display function is mounted.

FIG. 30 is a diagram that illustrates an example of an operation guidedisplay according to this embodiment. In the example illustrated in FIG.30, the projection-type display device 205 displays one display contenton a display screen 3000 projected onto a desk. In the display content,upward, downward, leftward, and rightward directions are as illustratedin the drawing. As an example of the guide display of theprojection-type display device 205, as illustrated in FIG. 30, aplurality of operation guide displays 3001, 3002, and 3003 are displayedat mutually-different angles for the display content. The example of theoperation guide display is a so-called on-screen display (OSD) and isdisplayed by a display control unit 111 to be superimposed on an inputsignal based on data stored in a storage unit, which is not illustratedin the drawing, arranged inside the projection-type display device 205under the control of the control unit 110.

In FIG. 30, not only the operation guide display 3001 displayed in thesame directions as the upward, downward, leftward, and rightwarddirections of the display content, the operation guide displays 3002 and3003 displayed in different directions are displayed. Accordingly, amongusers standing on the periphery of the desk, an operation method can beconveyed not only to a viewer viewing the display content on the frontside but also to viewers viewing the display content at angles otherthan the angle of the front side.

Particularly, as illustrated in FIG. 30, in a situation in which theprojection-type display device 205 is installed to one side A amongsides A to D of the rectangular display screen 3000, there is a lowpossibility that a viewer who can perform a gesture operation is presenton the side A. Here, in a case where viewers are present on three sidesB, C, and D other than the side A on which the projection-type displaydevice 205 is installed, and the display content is displayed in adirection in which the display content can be easily viewed from aposition of one side (for example, the side C) among them, when only theoperation guide display 3001 displayed in the same directions as theupward, downward, leftward, and rightward directions of the displaycontent is displayed, the viewers present on the other two sides (forexample, the sides B and D) do not know the operation method and arecompelled to take the trouble to move up to the position of the side C.

In contrast to this, in the projection-type display device 205 accordingto this embodiment, a plurality of operation guide displays can bedisplayed at mutually-different angles. As illustrated in FIG. 30, thedirections of the three sides B, C, and D may be displayed incorrespondence with the leftward/rightward directions of the pluralityof operation guide displays 3001, 3002, and 3003. By displaying as such,for example, all the viewers present on three sides B, C, and D sideother than one side A on which the projection-type display device 205 isinstalled can easily understand the operation method.

Here, the operation guide display may be a diagram illustrating anoperation gesture, a diagram illustrating an operation gesture, and adescriptive text thereof, a text describing an operation gesture, or ananimation illustrating an operation gesture. In addition, the operationguide display may be a combination thereof.

In addition, timing at which the operation guide display is displayedmay be timing at which an operation input of a remote controller buttonnot illustrated in the drawing is performed. Alternatively, it may beconfigured such that a small question mark is displayed on one corner ofthe display screen, and the operation guide display as illustrated inFIG. 30 is performed at timing at which viewer' finger touch or gestureoperation for the question mark is detected.

According to the operation guide display of this embodiment describedabove, an operation guide display having improved usability for viewerscan be performed.

Fourth Embodiment

Next, Embodiment 4 of the present invention will be described.

In the projection-type display device described in Embodiment 1 of thepresent invention, as described above, various screen operations can beperformed through user's gesture operations (pointing operations). Inthis embodiment, an example will be described in which, in addition tothe configurations and the functions of the projection-type displaydevices according to Embodiments 1, 2, and 3, a function for displayingdetailed information of a display video and a menu screen is mounted.

FIGS. 31A to 31D illustrate operations at the time of displayingdetailed information of a display video and a menu screen in a casewhere a projection-type display device 205 displays one display screen200 on a desk 204.

As an example of a gesture recognized when detailed information of adisplay video and a menu screen are displayed, a gesture in a state inwhich two fingers having different pointing directions are brought intocontact with each other as illustrated in FIG. 31A may be used. Here,two contact points do not need to be completely brought into contactwith each other but may satisfy a condition of having a predetermineddistance set in the design or less. In addition, as another example of agesture recognized when detailed information of a display video and amenu screen are displayed, a gesture in a state in which three fingersare brought into contact with a display surface as illustrated in FIG.31B may be used.

Here, the gestures illustrated in FIGS. 31A and 31B are examples, andany other gesture may be used as long as the gesture can bediscriminated from the other operations.

In this embodiment, in a case where the gesture illustrated in FIG. 31Aor 31B is detected, a control unit 110 of the projection-type displaydevice 205 determines a detailed information display operation anddisplays detailed information 3101 as illustrated in FIG. 31C.

In addition, as another example, in a case where the gesture illustratedin FIG. 31A or 31B is detected, the control unit 110 of theprojection-type display device 205 determines a menu screen displayoperation and, as illustrated in FIG. 31D, displays a menu screen 3102.

Furthermore, as another example, in a case where the gesture illustratedin FIG. 31A or 31B is detected, the control unit 110 of theprojection-type display device 205 determines an operation forperforming both the detailed information display and the menu screendisplay and may perform the display of the detailed information 3101illustrated in FIG. 31C and the display of the menu screen 3102illustrated in FIG. 31D together.

The positions of the display of the detailed information 3101illustrated in FIG. 31C and the display of the menu screen 3102illustrated in FIG. 31D may be set inside the display screen 200 or topredetermined neighboring positions. In addition, as another example,the detailed information 3101 or the menu screen 3102 may be displayedat a position corresponding to a finger's contact point of the gestureillustrated in FIG. 31A or a position (the position of a center fingeramong three fingers) at which the gesture illustrated in FIG. 31B isdetected.

As an example of the detailed information 3101, the detailed informationmay be characteristics (resolution at the time of input, a frame rate,interlaced or progressive, and the like) of a displayed video, and, in acase where supplementary information such as title information isincluded in an input video, the detailed information may be suchinformation. In addition, in case of a projection-type display devicethat includes a plurality of input terminals and can receive videoinputs from different input sources, the detailed information may beinformation used for identifying such an input source.

As an example of menu items displayed on the menu screen 3102, alanguage switching menu for a language used on the menu, an adjustmentmenu for image qualities such as the brightness, the contrast, and thelike of a display video, a trapezoidal correction process menu, and thelike may be displayed. In addition, in case of a projection-type displaydevice that includes a plurality of input terminals and can receivevideo inputs from different input sources, menu items used for switchingamong the input sources may be displayed.

Next, an example of a case where, after an operation for increasing thenumber of display screens described with reference to FIGS. 23A and 23Bis performed, an individual detailed information display or anindividual menu screen display is performed for each display screen willbe described with reference to FIGS. 32A and 32B.

FIGS. 32A and 32B illustrate a gesture for the operation of increasingthe number of display screens and a display example thereof asillustrated in FIGS. 23A and 23B, and the description thereof has beenpresented in Embodiment 1 and thus will not be presented here. FIG. 32Cillustrates an example in which the gesture illustrated in FIG. 31A isdetected at a position at which a finger's contact point is includedinside the display screen 202 out of a plurality of display screens 202and 203 generated by the operation of increasing the number of displayscreens. In this case, the control unit 110 of the projection-typedisplay device 205, as illustrated in FIG. 32D, performs an individualdetailed information display or an individual menu screen display forthe display screen 202 or a display combining these near the displayscreen 202 (or inside the screen).

In such a case, the individual menu screen illustrated in FIG. 32Ddisplays individual menu items for the display screen 202. For example,an image quality adjustment menu (the image quality adjustment for adisplay video displayed on the display screen 203 is not performed) fora display video displayed on the display screen 202 may be displayed. Inaddition, in case of a projection-type display device that includes aplurality of input terminals and can receive video inputs from differentinput sources, menu items used for switching (switching among the inputsources for a display video displayed on the display screen 203 is notperformed) among the input sources for a display video displayed on thedisplay screen 202 may be displayed. Furthermore, it is unnatural toperform a trapezoidal correction only for the display screen 202 that isone of the plurality of display screens, and thus, in the individualmenu display for the display screen 202, menu items used for thetrapezoidal correction may not be displayed or may be configured to bedisplayed but not to be operated.

Similarly, individual detailed information illustrated in FIG. 32Ddisplays individual detailed information for the display screen 202.Information used for identifying an input source and the like such asthe characteristics, title information, and the like of a videodisplayed on the display screen 202 may be displayed. In case of aprojection-type display device that includes a plurality of inputterminals and can receive video inputs from different input sources,switching among the input sources can be performed on the menu screenfor the display screen 202 illustrated in FIG. 32D. At this time, sincethe characteristics, the title information, the input sources, and thelike of videos displayed on the display screens 202 and 203 aredifferent from each other, it is advantageous to enable a viewer toindividually check the detailed information of each display screen.

In the examples illustrated in FIGS. 32A to 32D, as the gesture used forthe individual detailed information display or the individual menuscreen display, while the example of the gesture illustrated in FIG. 31Ais used, the gesture illustrated in FIG. 31B may be used.

Next, an example of a case where, after the rotation operation of thedisplay screen described with reference to FIGS. 22A, 22B, 25A, and 25Bis performed, an individual detailed information display or anindividual menu screen display for each display screen is performed willbe described with reference to FIGS. 33A to 33D.

FIGS. 33A and 33B illustrate a gesture for the rotation operation of thedisplay screen and a display example thereof as illustrated in FIGS. 22Aand 22B, and the description thereof has been presented in Embodiment 1and thus will not be presented here. FIG. 33C illustrates an example inwhich the gesture illustrated in FIG. 31B is detected at a position atwhich a finger's contact point is included inside the display screen 203in a state in which one display screen 203 out of a plurality of displayscreens 202 and 203 is rotated by the rotation operation of the displayscreen. In this case, the control unit 110 of the projection-typedisplay device 205, as illustrated in FIG. 33D, performs an individualdetailed information display or an individual menu screen display forthe display screen 203 or a display combining these near the displayscreen 203 (or inside the screen). At this time, the control unit 110performs the individual detailed information display or the individualmenu screen display at an angle corresponding to the display angle ofthe display screen 203. The content of the individual detailedinformation display or the individual menu screen display for thedisplay screen 203 is similar to that of the display screen 202illustrated in FIGS. 32A to 32D, and the description thereof will not berepeated.

For a rotated displayed screen, there is a high possibility that aviewer present at a position for easy viewing the display screenperforms an individual detailed information display or an individualmenu screen display through a gesture operation. Accordingly, asillustrated in FIG. 33D, by performing the individual detailedinformation display or the individual menu screen display at an anglecorresponding to the angle of the display screen, there is an advantagethat a viewer can easily operate the viewing screen.

In the examples illustrated in FIGS. 33A to 33D, as the gesture used forthe individual detailed information display or the individual menuscreen display, while the example of the gesture illustrated in FIG. 31Bis used, the gesture illustrated in FIG. 31A may be used.

Next, an example will be described with reference to FIGS. 34A to 34D inwhich, in a case where a plurality of display screens are displayed, notan individual menu for each screen but a whole menu for all the displayscreens at its targets is displayed. In the example illustrated in FIGS.34A to 34D, as a gesture used for displaying the whole menu, similar tothe examples illustrated in FIGS. 32A to 32D and FIGS. 33A to 33D, whilea gesture similar to that illustrated in FIG. 31A or 31B is used, thedetection position of the gesture is different from that of the examplesillustrated in FIGS. 32A to 32D and FIGS. 33A to 33D. For example, in acase where a finger's contact point of the gesture illustrated in FIG.31A is detected outside the plurality of display screens, a gesture forstarting the whole menu display is determined. Similarly, in a casewhere three fingers in a gesture similar to that illustrated in FIG. 31Bare detected outside the plurality of display screens, a gesture forstarting the whole menu display is determined. Hereinafter, morespecific description will be presented with reference to drawings.

In the example illustrated in FIG. 34A, a projection-type display device205 displays a display screen 202 and a display screen 203 on a desk204. Here, the gesture illustrated in FIG. 31A is detected, and afinger's contact point is detected at a position not inside the displayscreen 202 and the display screen 203. In this case, a control unit 110of the projection-type display device 205, as illustrated in FIG. 34B,displays a whole menu 3401 by controlling a display control unit 111. Inaddition, whole detailed information 3402 that is detailed informationfor the whole screen may be displayed.

Here, as menu items of the whole menu 3401, a menu of whole imagequality adjustment that simultaneously performs image quality adjustmentfor the display videos displayed on all the display screens may bedisplayed. In addition, menu items used for switching among languagesused on the whole menu may be displayed. Furthermore, a menu item of thetrapezoidal correction that cannot be used in the individual menudisplay of each individual screen may be configured to be usable on thewhole menu 3401.

As an example of the whole detailed information 3402, there is thenumber of display screens or the like. In addition, in case of aprojection-type display device that includes a plurality of inputterminals and can receive video inputs from different input sources, aplurality of display screens and a list of input sources may bedisplayed.

In addition, in the example illustrated in FIG. 34A, the gestureillustrated in FIG. 31B may be used in place of the gesture illustratedin FIG. 31A.

Next, in the example illustrated in FIG. 34C, the projection-typedisplay device 205 displays the display screen 202 and the displayscreen 203 on the desk 204 and displays the display screen 203 at anangle different from that of the display screen 202 through a rotationprocess. Here, the gesture illustrated in FIG. 31A is detected, and allthe finger's contact points are detected at a position not inside thedisplay screen 202 and the display screen 203. In this case, the controlunit 110 of the projection-type display device 205, as illustrated inFIG. 34D, displays the whole menu 3401 by controlling the displaycontrol unit 111. In addition, as in the display example illustrated inFIG. 34D, at this time, the individual detailed information 3403 and3404 may be displayed for a plurality of display screens.

Here, in the example illustrated in FIG. 33D, when the individualdetailed information is displayed, the individual detailed informationis displayed at an angle corresponding to the display angle of theindividual display screen. In contrast to this, as in FIG. 34D, in acase where the whole menu 3401 is displayed, it is preferable that theindividual display screen is displayed at the same angle as the displayangle of the whole menu 3401 regardless of the display angle of theindividual display screen 203 so as to be easily seen by an operator ofthe whole menu 3401.

In the description presented above, in the example illustrated in FIG.34B, while the whole detailed display is performed together with thewhole menu display, and, in the example illustrated in FIG. 34D, theindividual information display is performed together with the whole menudisplay, any one of the displays may be performed. In addition, only thewhole menu may be displayed. Furthermore, all the whole menu display,the whole detailed display, and the individual detailed informationdisplay may be performed.

In addition, on the menu screen (the individual menu screen and thewhole menu screen) described in this embodiment, a menu item used forsetting On/Off of each gesture recognition function described in eachembodiment may be arranged. For example, all the gesture recognitionfunctions may be turned off. In such a case, the projection-type displaydevice 205 is operated based on an operation signal corresponding to abutton operation for a remote controller or a main body. The recognitionprocess for a one-finger moving gesture illustrated in FIG. 21A, therecognition process for a one-finger rotation gesture illustrated inFIG. 21B, and the like may be configured to be individually turned off.It is effective in a case where there is a possibility of erroneousrecognition at the time of description using a bar or the like.

In addition, in a rotation operation, a menu item used for selectingrecognition of the one-finger rotation gesture illustrated in FIG. 21Bor the two-finger rotation gesture illustrated in FIGS. 25A and 25B maybe displayed. As above, depending on the situations, there are caseswhere it is advantageous to individually set On/Off of each gesturerecognized by the projection-type display device 205. In such cases, forexample, On/Off may be configured to be settable for all the gesturesillustrated in FIG. 21A and subsequent diagrams on the menu screen. Incase of the individual menu screen, the On/Off setting of such gesturesmay be applied to only a corresponding display screen.

According to the detailed information display and the menu display ofthis embodiment described above, a detailed information display and amenu display having improved usability for a viewer can be performed.

REFERENCE SIGNS LIST

-   100 Camera-   101, 102 Illumination-   104 Shadow area extracting unit-   105 Feature point detecting unit-   106 Approaching degree detecting unit-   107 Contact point detecting unit-   108 Contour detecting unit-   109 Direction detecting unit-   110 Control unit-   111 Display control unit-   112 Drive circuit unit-   113 Input terminal-   114 Input signal processing unit-   115 Projection unit-   201 Projection-type display device-   202, 203 Display screen-   204 Projection object-   205 Projection-type display device provided with operation detecting    function-   3 User-   30, 31, 32 Finger-   401, 402, 403, 404, 405 Shadow-   501, 502 Contour line-   601, 602 Feature point-   700, 701, 702, 703 Pointing direction-   800 Pointer-   P Contact point

The invention claimed is:
 1. A projection-type video display devicecomprising: a video projector that projects a video; at least one lightsource that emits illumination light different from that of theprojected video onto a surface at least partly overlapping a videoprojection surface onto which the video projector projects the video; animage capture device that performs imaging using reflected light of theillumination light; and a control system that is communicatively coupledto the video projector, the at least one light source and the imagecapture device, wherein the control system: detects at least a positionat which an operator's finger is brought into contact with the videoprojection surface based on a captured image acquired by the imagecapture device; and performs a plurality of operations that change atleast one partial video included in the video projected by the videoprojector based on a result of the detection acquired by the controlsystem, wherein, the plurality of operations include a first operationand a second operation; wherein the first operation comprises: dividinga first partial video of video projected by the video projector into aplurality of separate display screens, on a condition that theoperator's finger is detected to move into the first partial video fromoutside of the first partial video and traverses the partial video whilebeing brought into contact with the video projection surface; andallowing each of new partial video contents of the separate displayscreens to be independently adjustable through interaction of each ofthe separate display screens, and wherein the second operationcomprises: on a condition that the operator's finger is detected to movefrom a start position in a second partial video while being brought intocontact with the video projection surface, moving the second partialvideo of video projected by the video projector.
 2. The projection-typevideo display device of claim 1, further comprising a input signalprocessor, wherein the video projector projects a list of input sourcesto be inputted to the input signal processor for selecting the contentof the separate display screens.
 3. The projection-type video displaydevice of claim 1, wherein the control system detects the position atwhich the operator's finger is brought into contact with the videoprojection surface based on a feature of at least one shadow found inthe captured image.
 4. The projection-type video display device of claim1, wherein the at least one light source includes two light sources, andthe control system detects the position at which the operator's fingeris brought into contact with the video projection surface based onfeatures of two different shadows cast by the operator's finger on thevideo projection surface.